Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
Specific knowledge learning based on knowledge distillation
Zhaoxia DAI, Yudong CAO, Guangming ZHU, Peiyi SHEN, Xu XU, Lin MEI, Liang ZHANG
Journal of Computer Applications    2021, 41 (12): 3426-3431.   DOI: 10.11772/j.issn.1001-9081.2021060923
Abstract367)   HTML25)    PDF (648KB)(180)       Save

In the framework of traditional knowledge distillation, the teacher network transfers all of its own knowledge to the student network, and there are almost no researches on the transfer of partial knowledge or specific knowledge. Considering that the industrial field has the characteristics of single scene and small number of classifications, the evaluation of recognition performance of neural network models in specific categories need to be focused on. Based on the attention feature transfer distillation algorithm, three specific knowledge learning algorithms were proposed to improve the classification performance of student networks in specific categories. Firstly, the training dataset was filtered for specific classes to exclude other non-specific classes of training data. On this basis, other non-specific classes were treated as background and the background knowledge was suppressed in the distillation process, so as to further reduce the impact of other irrelevant knowledge on specific classes of knowledge. Finally, the network structure was changed, that is the background knowledge was suppressed only at the high-level of the network, and the learning of basic graphic features was retained at the bottom of the network. Experimental results show that the student network trained by a specific knowledge learning algorithm can be as good as or even has better classification performance than a teacher network whose parameter scale is six times of that of the student network in specific category classification.

Table and Figures | Reference | Related Articles | Metrics